Recently, portable communication terminals, such as cellular phones, have become quite popular. Those portable communication terminals are now equipped with not only the conversation function by speech, but also other various functions, e.g., transmission/reception of electronic mail, access to the Internet, and reception of a still image or a mobile image picked up by a camera on the conversation partner side. Nowadays, in addition to the cellular phones, two-way video communication systems connecting two distant places to each other via a video and voice communication network, such as a videoconference system and a videophone, have also expanded their use.
In some of the cellular phones and the two-way video communication systems, an image of the conversation partner is displayed on an image display disposed near a camera for taking a picture of the user's face, etc. The user of such a cellular phone and two-way video communication system usually makes conversation while looking at the image, e.g., the face of the conversation partner, displayed on the image display.
When an image pickup device, e.g., a camera, is mounted in a relatively small unit such as a cellular phone, the image pickup device is disposed outside an image display unit away in the right-and-left direction or the vertical direction. In such a case, even if the image display unit has a small size of, e.g., about 50 mm square, the eyes (line of sight) of the picked-up face image are not directed toward the image pickup device so long as the user looks at the image display unit where the conversation partner is displayed. As a result, that type of equipment displays, on the image display unit, the face image not matched in the line of sight. In that type of equipment, therefore, the user makes conversation with the conversation partner while looking at the face image not matched in the line of sight, thus resulting in the problems that unnaturalness is not avoidable and a realistic conversation impression is lost.
More practically, such a disadvantage can be explained as follows with reference to FIGS. 27A, 27B, 28A and 28B. FIGS. 27A, 27B, 28A and 28B show examples in which the direction of line of sight is not matched with respect to a face image on the image display unit of the terminal. Each of these examples shows an image picked up when a camera is disposed at a position 65 mm away from the center of the image display unit and the user looks at the center of the image display unit while his face is positioned at a distance of about 25 cm.
FIGS. 27A and 27B each illustratively shows an image that is picked up by a camera disposed leftward or rightward of the image display unit being about 50 mm square and is displayed on the image display unit. More specifically, FIG. 27A shows an image picked up by a camera that is disposed on the left side as viewed toward the image display unit, and FIG. 27B shows an image picked up by a camera that is disposed on the right side as viewed toward the image display unit. As seen from those drawings, the eyes of the face in each camera image are not directed toward the user from the image display unit, and hence the face image is unnatural in conversation.
Similarly, FIGS. 28A and 28B each illustratively shows an image that is picked up by a camera disposed above or below the image display unit being about 50 mm square and is displayed on the image display unit. More specifically, FIG. 28A shows an image picked up by a camera that is disposed on the upper side of the image display unit, and FIG. 28B shows an image picked up by a camera that is disposed on the lower side of the image display unit. In any of these cases, the eyes of the face in each camera image are not directed toward the user from the image display unit, and hence the face image is unnatural in conversation.
To overcome that unnaturalness, various eye-to-eye matching techniques have been proposed in the past.
According to one of those techniques, a small half mirror substantially in the form of a flat plate is disposed on the surface of an image display unit so that an image displayed on the image display unit passes the small half mirror for presenting display, while an image of the user is received by an image pickup camera which captures light reflected by the surface of the small half mirror. This technique can provide eye-to-eye matching display because the eyes of the user looking at the image display unit are matched with an optical axis of light incident upon the small half mirror before the light is reflected by the mirror.
According to another technique, by way of example, an image display unit having a light transmissible structure is provided and a camera is disposed on the backside of the image display unit relative to the user. This technique repeatedly brings the image display unit into a display state and a transmitted state in a time-division manner, and causes the camera to pick up an image of the user when the image display unit is in the transmitted state. Further, according to this technique, during a period other than the transmitted state, a desired video signal is sent to the image display unit and an image of the conversation partner is displayed. This technique can also provide eye-to-eye matching display because the direction of light emerging from the image display unit is matched with an optical axis of light passing through the image display unit and entering the camera disposed on the backside of the image display unit.
As still another technique, there is known a display and image pickup device as disclosed in Japanese Unexamined Patent Application Publication No. 4-167690. In this display and image pickup device, a number of small holes are formed all over the surface of an image display unit. One end of an optical fiber is positioned to face each of the small holes, and the other end of each optical fiber is connected to a camera. This display and image pickup device can also provide eye-to-eye matching display because the positional relationship between the ends of the optical fibers positioned to face the respective small holes and the image display unit is not offset.
While the above-mentioned known techniques are realized by aligning the image display unit with the optical axis of the image pickup device, it is further known, as a different type eye-to-eye matching technique, to synthesize eyes themselves by the use of computer graphics, as described in, e.g., “Tsuyoshi Yamaguchi et al., “Proposal for eye-to-eye matching technique in videoconference”, Proceedings of Sixth Image Sensing Symposium, p267-p272, 2000”.
In addition, a try to realize the eye-to-eye matching in a stereoscopic way is lately proposed (see, e.g., Japanese Unexamined Patent Application Publication No. 10-75432).
The Japanese Unexamined Patent Application Publication No. 10-75432 discloses a stereoscopic videophone in which image pickup units, constituted as cameras, and an image display unit are provided in a housing put on a table. The image display unit is constituted by using image splitter type three-dimensional liquid crystal display devices without spectacles, and the cameras are disposed in left-side and right-side positions of the image display unit.
Also, the Japanese Unexamined Patent Application Publication No. 10-75432 mentions that images picked up by the two cameras disposed in the left-side and right-side positions of the image display unit are selectively synthesized through fusion, whereby a front face image capable of providing a pseudo stereoscopic view can be obtained and users can make conversation in an eye-to-eye matching state.
In the above-mentioned various techniques realized by aligning the image display unit with the optical axis of the image pickup device, the image pickup device is arranged in some positional relationship relative to the image display unit.
More specifically, the above-described technique using the small half mirror requires the image pickup device to be positioned in the reflecting direction in which the light is reflected by the surface of the small half mirror. Also, the above-described technique of picking up a face image by using the light passing through the image display unit requires the camera to be disposed on the backside of the image display unit. To realize those techniques, therefore, the camera, etc. must be held by an apparatus having an increased overall size, and a difficulty exists in realizing those techniques in cellular phones, for example.
Also, in the device such as described in the above-cited Japanese Unexamined Patent Application Publication No. 4-167690 wherein optical fibers are attached to small holes formed in an image display unit, because of a difficulty in assembly steps of forming the small holes in the image display unit and inserting the ends of the optical fibers into the respective small holes in one-to-one relation, the product price is significantly increased.
Further, the technique of synthesizing the eyes of the communication partner by utilizing computer graphics, as described in the above-cited paper “Proposal for eye-to-eye matching technique in videoconference”, dose not cause the above-mentioned problem attributable to the mount position of the image pickup device. At the current technology level of computer graphics, however, a synthesized image is still far from an actual one and unnaturalness in the line of sight of the communication partner cannot be yet completely eliminated.
Further, in the stereoscopic videophone disclosed in the above-cited Japanese Unexamined Patent Application Publication No. 10-75432, if the image display unit has a size of about 14 inches, for example, the distance from the left side to the right side of the image display unit in the lateral direction is about 30 cm. Therefore, when two cameras are disposed in positions not interfering with the image display unit, a relatively large shift occurs between two picked-up images. Consequently, the videophone has the problem that when those two images are used, as they are, for stereoscopic image, a parallax is too increased to synthesize such a double image, as it is, through fusion. Even if the double image can be synthesized through fusion, the resulting display imposes a burden on the user's eyes and makes the user feel fatigue.
The present invention has been accomplished in view of the state of the art set forth above, and its object is to provide, in relation to a portable information processing apparatus such as a portable communication terminal, an information processing apparatus, an information processing system, and a conversation partner display method, which can realize conversation in a natural eye-to-eye matching state with a conversation partner. Another object of the present invention is to provide an information processing apparatus, an information processing system, and a conversation partner display method, which can avoid the disadvantage resulting from a too large parallax between two picked-up images and which can realize conversation in a natural eye-to-eye matching state with a conversation partner.